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Abstract 

The  nations  of  the  world  are  moving  toward  performance  based  building  code  standards  which  will  establish 
a level  of  safety  or  risk  rather  then  the  traditional  prescriptive  codes  which  specify  the  performance  of 
components.  Performance  evaluations  can  then  use  trade  offs  between  many  factors  to  provide  the  required 
level  of  safety.  Computer  models  are  the  means  to  ascertain  the  performance  of  buildings  built  with  new 
materials  and  new  contents.  As  these  models  progress  and  become  more  entrenched  in  the  regulatory 
system,  it  is  paramount  there  be  a continual  effort  to  insure  their  validity.  The  accuracy  of  the  models  of 
individual  phenomena  is,  and  should  be,  addressed  during  development.  However,  the  interaction  of  various 
parts  of  the  system  are  not  always  well  understood.  This  paper  is  part  of  the  continuing  effort  to  test  the 
complete  system  model  with  full  scale  and  real  scale  tests  and  experiments.  The  International  Standards 
Organization  (ISO),  together  with  the  Conseil  International  du  Batiment  (CIB),  is  establishing  a framework 
for  deciding  on  the  appropriateness  of  a model  to  meet  the  requirements  of  those  who  wish  to  use  them  in 
predicting  the  environment  in  a building.  This  paper  discusses  the  status  of  this  work  and  lays  out  the  time 
table  for  the  completion  of  this  effort,  leading  to  a proposal  for  an  ISO  standard. 


Introduction 

Computer  modeling  of  physical  phenomena  is  becoming  an  increasingly  important  tool 
for  understanding  the  physical  world.  The  driving  forces  are  the  increased  speed  and  availability 
of  computers  and  an  increased  knowledge  base  of  the  physical  world.  These  factors  have  led  to 
improvements  such  that  model  predictions  are  being  used  for  actual  design  decisions.  Zone 
models  are  the  present  means  of  choice  for  understanding  and  predicting  fire  growth  and  smoke 
spread  in  compartmented  structures.  In  the  past,  such  models  have  been  used  largely  to 
determine,  after  the  fact,  what  occurred  in  specific  fires.  As  confidence  in  the  capability  of  such 
techniques  grows,  many  nations  are  moving  toward  replacing  much  of  their  bench  scale  testing 
of  components  with  computer  modeling  of  whole  systems[l]. 

Several  attempts  have  been  made  to  answer  partially  the  question  of  how  close  the 
models  are  to  reality.  As  noted  in  Peacock,  Jones,  and  Bukowski[2],  to  date  the  method  has  been 
to  examine  plots  of  experimental  data  and  model  predictions.  Then  some  qualitative  and  often 
vague  statement  on  the  "goodness"  of  the  prediction  is  made.  No  method  exists  for  making 
quantitative  estimates  of  error  in  predictions  and  there  are  many  difficulties  in  developing  such  a 
method[3].  The  problem  concerns  comparing  curves  of  variables  plotted  against  time.  A method 
of  comparing  two  curves  and  giving  a simple  report  on  how  they  fit  each  other  doesn't  exist. 
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Until  some  method  exists,  quantitative  comparisons  can  not  be  done.  Although  we  cannot  yet 
provide  such  a quantitative  comparison,  it  is  possible  to  provide  graphically  a qualitative 
analysis.  As  is  the  practice,  predictions  have  been  plotted  against  measured  values.  This  gives  a 
visual  sense  of  errors  and  any  trends.  The  graphs  used  are  included  to  allow  examination  of  the 
data  that  leads  to  these  conclusions. 


Overall  Project  Objective 

Our  objective  in  the  project  is  rather  broad:  To  develop  and  promote  an  international 
protocol  to  assess  the  predictive  capabilities  and  usability  of  computer-based  fire  models  that 
includes  the  experimental,  statistical,  and  analytical  techniques  to  implement  them.  There  are  an 
number  of  areas,  however,  where  significant  technical  issues  must  be  resolved  before  such  a 
process  is  workable. 


Issues  Which  Must  be  Addressed 

It  is  critical  that  the  process  of  model  evaluation  addresses  areas  of  primary  interest  to  the 
users  of  the  models.  With  this  in  mind,  a number  of  areas  of  interest  to  model  users  can  have  an 
impact  on  future  efforts  in  fire  model  evaluation: 

• Classes  and  uses  of  models  - What  types  of  models  are  available  (zone  models,  field  models,  special  purpose 
models)  and  how  are  they  being  used  (design,  fire  recreation,  litigation)?  Also  included  is  the  intended  domain 
of  models  (and  typical  use  which  may  be  different).  Several  summaries  exist,  which  list,  but  do  not  evaluate 
these  models. 

• Scenarios  for  use  in  model  evaluation  - What  fire/building  scenarios  are  important  to  the  users  of  the  models? 
The  answer  to  this  should  include  sufficient  information  to  allow  appropriate  experiments  or  simulations  to  be 
conducted  (building  description,  design,  openings  and  connections,  materials;  contents,  ignition  scenario, 
environmental  conditions).  Any  process  of  model  evaluation  should  concentrate  on  areas  which  tests  the  limits 
of  the  models  rather  than  areas  which  most  models  can  be  expected  to  successfully  predict. 

• Developing  and  gaining  acceptance  of  an  evaluation  method  - How  can  we  insure  that  whatever  process  is 
used  to  evaluate  a model  is  supported  by  the  users  and  developers  of  the  models? 

• Technical  issues  - Although  we  should  build  on  extensive  available  experimental  data,  it  is  likely  that  there  are 
areas  which  are  yet  to  have  been  studied  in  detail.  These  include  non-residential  fire  scenarios,  statistical  and 
analytical  techniques  for  comparing  model  and  experimental  data,  uncertainty  in  model  and  experimental  results, 
and  model  sensitivity. 


Important  Elements  of  a Protocol  for  Model  Evaluation 

Once  the  most  important  models  and  scenarios  are  defined,  there  are  multiple  approaches 
to  model  evaluation:  evaluation  of  the  model’s  theoretical  basis  by  peer  review,  evaluation  of  the 
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implementation  and  usability  by  model  users,  and  evaluation  of  the  overall  predictive  capability 
of  the  resulting  model  by  comparison  of  the  model  with  experimental  results.  A summary  of 
these  may  provide  a starting  point  for  discussions  towards  a widely  accepted  protocol: 

• Peer  review  of  theoretical  basis  of  models 

• Usability  and  practicality  of  models 

• Comparison  of  model  predictions  with  experimental  results 

Sources  and  quality  of  data  for  comparisons 
Measurement  and  measurement  system  requirements 
Uncertainty  in  experimental  results 
Peer  review  of  experimental  results 

Methodology  for  transforming  measured  data  into  values  which  may  be  compared 
with  model  predictions 

Sensitivity  analysis  of  the  experimentally-based  "model  predictions” 

Blind  nms  of  the  models 
Statistical  and  analytical  comparisons 

Peer  review  of  theoretical  basis  - The  theoretical  basis  of  the  model  should  be  reviewed  by  a 
group  of  recognized  experts  fully  conversant  with  the  chemistry  and  physics  of  fire  phenomena. 
The  reviewers  should  judge  whether  there  is  sufficient  scientific  evidence  in  the  open  scientific 
literature  to  justify  the  approaches  and  assumptions  being  used.  Primarily,  this  group  would  be 
comprised  of  researchers  in  the  field. 

Issues  to  be  resolved  include  who  should  participate  in  the  groups,  who  is  willing  to  pay  for  the 
effort,  what  organization  should  be  responsible  for  the  group,  and  how  can  the  group's 
independence  be  ensured  with  a group  composed  of  both  model  developers  and  users.  Key  to  the 
success  of  such  an  effort,  however,  is  some  agreement  on  a process  and  criteria  for  submission 
and  evaluation  of  models  that  is  acceptable  to  the  majority  of  the  model  developers  and  users. 
Such  a process  is  critical  to  avoid  continual  philosophical  arguments  of  the  "best”  model 
phenomena. 

Practicality  and  usability  - In  addition  to  peer  review  of  the  physics  of  a model,  the  ease  with 
which  data  can  be  input  to  the  model  and  the  outputs  of  the  model  examined  can  have  an  impact 
on  the  appropriate  and  correct  use  of  a model.  Like  the  peer  review  of  the  theoretical  basis  of 
models,  a group  of  model  users  could  review  the  documentation  and  use  to  judge  the  level  of 
expertise  necessary  to  use  different  models. 

Comparison  of  model  predictions  with  experimental  results  - User  acceptance  of  predictive 
fire  models  is  enhanced  when  the  predictions  of  the  model  matches  the  user's  experience.  This 
may  come  from  successful  recreation  of  actual  fire  incidents  or  through  comparison  of  the 
predictions  of  a model  with  data  gathered  experimentally.  In  addition,  verifying  that  the  entire 
model  satisfactorily  predicts  the  course  of  a fire  by  comparing  its  output  against  the  actual  full- 
scale  fire,  ensures  that  the  inevitable  assumptions  and  errors  in  the  individual  submodels  do  not 
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combine  to  produce  incorrect  predictions.  Program  predictions  should  be  made  without  reference 
to  the  experimental  data  to  be  used  for  the  comparison.  Of  course,  this  restriction  does  not 
include  required  input  data  that  may  have  been  obtained  by  bench  or  larger  scale  tests.  No 
attempt  to  adjust  a fit  between  the  measurements  and  the  predictions  should  be  made. 

The  quality  of  experimental  data  must  also  be  assured.  Any  experimental  data  used  for  model 
evaluation  must  be  sufficiently  documented  and  reviewed  to  insure  that  the  experiments  were 
conducted  appropriately,  that  the  facilities,  instrumentation,  and  experimental  techniques  were 
appropriate  to  the  scenario  investigated,  that  the  experimental  data  is  consistent,  and  that  the 
experiments  have  been  sufficiently  characterized  to  allow  simulation  by  the  models  without 
actual  reference  to  the  experimental  data.  Physical  measurements  of  compartments,  connections, 
leakage,  and  construction  materials  and  techniques  must  be  adequately  described  to  allow 
accurate  simulation.  Uncertainties  in  the  measurements  and  measurement  techniques  themselves 
should  be  determined  in  a systematic  and  logical  manner. 

Model  documentation  - In  order  to  evaluate  the  predictive  capability,  practicality,  and  usability 
of  any  model,  sufficient  documentation  must  be  provided  by  the  model  developer  to  enable  an 
independent  review  of  the  theoretical  assumptions  and  mathematical  techniques  used  in  the 
model.  Such  documentation  of  a computer  model  is  the  primary  means  for  the  model  developer 
to  communicate  the  underlying  physics,  chemistry,  and  assumptions  of  the  model. 

Sensitivity  analysis  - Nearly  all  fire  models  are  deterministic.  However,  uncertainties  in  model 
inputs  lead  to  often  overlooked  uncertainties  in  the  model  results.  Model  predictions  may  be 
sensitive  to  uncertainties  in  input  data,  to  the  level  of  rigor  employed  in  modeling  the  relevant 
physics  and  chemistry,  and  to  use  of  numerical  treatments.  In  general,  we  need  to  develop  a 
procedure  which  identifies  the  important  steps  and  minimum  requirements  for  evaluation  of  the 
sensitivity  of  a range  of  fire  models  for  consideration  by  national  and  international  consensus 
standards  organizations,  and  to  identify  and  review  methods  for  sensitivity  analysis  which  may 
be  appropriate  for  evaluation  of  computer  based  fire  models. 

• Organize  a working  group  to  develop  a consensus  on  a set  of  fire/building  scenarios  of 
interest  to  model  users. 

• Test  a range  of  available  fire  models  with  these  scenarios  (once  the  scenarios  are 
adequately  characterized)  to  refine  the  range  over  which  the  models  should  be  evaluated. 

• Develop  a consensus  document  on  a protocol  for  model  evaluation.  The  existing  ASTM 
guide  can  provide  a starting  point.  We  are  currently  working  to  expand  on  this  guide  and 
would  welcome  cooperation.  ISO  CD  13389  is  now  being  circulated. 
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The  CIB  W14  Project  on  Model  Verification 


An  assessment  procedure  for  the  evaluation  of  deterministic  fire  development  simulation 
methods  has  been  proposed.  Detailed  planning  is  made  for  the  first  phase  consisting  of  blind, 
semiblind  and  open  numerical  simulations  followed  by  a common  comparison  of  simulation  data 
with  experimental  data.  An  open  comparison  report  is  produced  for  every  scenario  of  the  round 
robin. 

This  proposal  has  been  prepared  within  a CIB  W14  sub-group  'Assessment  and  verification  of 
computer  codes  for  predicting  fire  development  and  smoke  movement'.  The  members  of  the 
group  are:  Michel  Curtat,  Centre  Scientifique  et  Technique  du  Batiment,  France,  Reinhold 
Dobbemack,  IBMB  der  Technischen  Universitat  Braunschweig,  Germany,  Walter  W.  Jones, 
Building  and  Fire  Research  Laboratory,  NIST,  USA,  and  Olavi  Keski-Rahkonen  (chair),  VTT 
Building  Technology,  Finland.  The  proposal  was  discussed  January  12,  1995  at  the  CIB  W14 
plenary  meeting  at  Espoo  and  accepted  as  rules  for  the  first  rounds  of  the  code  assessment  round 
robin. 


Although  numerous  efforts  to  compare  fire  models  with  fire  experiments  have  been 
published,  systematic  validation  of  the  plethora  of  existing  fire  codes  [4]  is  insufficient.  This 
deficiency  has  become  critical  by  the  introduction  of  performance  based  codes  in  several 
countries  which  place  heavy  emphasis  on  numerical  fire  simulations.  Therefore,  designers, 
authorities,  and  the  end  users  of  buildings,  who  may  not  be  knowledgeable  about  fire  simulation, 
should  be  given  guidance  on  which  codes  to  use  and  on  the  limits  of  the  models.  There  is 
anecdotal  information  that  the  codes  could  predict  whatever  you  want.  Currently  it  is  left  up  to 
the  skills  of  the  applicant  to  ensure  the  range  of  validity  of  a computer  code  applied  to  a 
particular  problem. 

Following  several  initiatives  CIB  W14  has  set  up  a prenormative  task  group[5]  to  plan 
and  organize  a validation  program,  which  is  intended  to  provide  unbiased  evaluations  of  different 
deterministic  fire  development  codes  and  a means  to  test  the  usability  of  these  models.  Primarily, 
this  evaluation  is  intended  to  include  zone  model  based  codes  due  to  existing  experimental 
information,  but  other  types  of  codes,  if  offered  to  participate  in  the  round  robin,  are  not 
excluded. 

Project  Objective 

The  objectives  of  this  group  are 

(1)  to  increase  confidence  in  the  use  of  fire  models  as  a tool  for  fire  safety  engineering, 
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(2)  to  support  ISO/TC92/SC4  in  its  effort  to  produce  a document  on  assessment  and  verification  of  calculation 
models  (WD13389), 

(3)  to  consider  all  aspects  of  code  evaluation,  including  physics,  numerics,  documentation,  use  of  the  codes,  and 
availability  of  appropriate  data  for  the  selected  scenarios,  and 

(4)  to  carry  out  a round  robin  project  on  deterministic,  numerical,  fire-simulation  computer  codes  and  experiments 
for  model  evaluation. 

These  definitions  are  slightly  modified  from  those  given  in  [4,  5]  due  to  opinions  presented 
during  discussion  within  the  group. 

The  objectives  of  the  evaluation  are  so  wide,  that  it  will  require  several  years  to  reach 
them.  It  is  feasible  to  plan  in  a detailed  way  only  a project  for  the  first  phase  consisting  of  two 
simple  scenarios.  Altogether  it  is  foreseen  that  at  least  ten  different  scenarios  should  be 
considered  to  assess  a code  to  the  extent  required  for  fire  safety  engineering.  Preliminarily,  these 
scenarios  could  be: 

A:  Single  plume  under  a hood 

B:  Single  room  with  a door  opening 

C:  Single  room  with  a door  opening  into  a corridor 

D:  A floor  in  a hotel  and/or  in  a health  care  facility 

E:  Atrium  and  a room  opening  into  an  atrium 

F:  Shopping  mall 

G:  Staircase  in  a multifloor  building 

H:  Very  large  room 

I:  Underground  space,  room  ventilated  only  from  above 

Compartment  scenarios  include  as  subscenarios  different  fire  load  configurations  and 
boundary  conditions  (natural  versus  mechanical  ventilation).  Scenarios  with  burning  walls, 
complicated  fire  loads  as  well  as  inclusion  of  active  extinction  are  left  to  the  later  part  of  the 
program  because  the  physical  basis  is  still  under  development. 

Project  Schedule 

Assessment  consisting  of  different  tasks  is  carried  out  as  field  work  at  different 
participating  institutions,  and  as  expert  work  within  the  CIB  W14  task  group.  These  are  indicated 
below  by  a letter  after  the  task  number.  Calculations  relative  to  experimental  data  are  carried  out 
as  blind,  semiblind  or  open.  The  final  report  is  intended  to  provide  a measure  of  the  validity  of 
the  data  used  for  comparison,  and  the  process  used  for  the  evaluation.  For  the  first  phase  of  the 
project,  the  proposed  schedule  is  as  follows: 

Project  program  accepted  (January  13,  1995). 

Identification  of  potential  participants  into  the  first  round  of  the  round  robin  (March  1995). 

Data  for  the  first  fire  scenarios  (standard  problems)  (March  1995) 
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(A)  smoke  filling  of  a hood  with  an  analytic  solution  to  test  numerics,  and 

(B)  three  sizes  of  compartments  with  a vertical  opening 

© worked  out  into  a standard  FDMS  format  [3],  and  design  reports  written. 

Invitations  to  participation  into  the  first  round  of  the  robin  sent  out  (April  1995) 

Announcement  of  acceptance  fulfilling  preconditions  (August  1995) 

Filing  the  blind  calculation  report  on  one-compartment  fire  (B).  In  addition  to  illustrating  the  comparability  of  models  in 
actual  end-use  conditions,  this  will  test  the  ability  of  the  model  users  to  develop  appropriate  input  data  for  the  models 
(ultimate  deadline  December  31,  1995) 

Deadline  for  peer  reviews  of  the  codes  (December  1995) 

Invitation  to  calculation  of  the  one-compartment  fire  with  given  rate  of  the  heat  release  (B),  (January  1996).  As  a follow- 
on  to  the  blind  calculation,  this  test  provides  a more  careful  comparison  of  the  underlying  physics  in  the  models  with  a 
more  completely  specified  scenario. 

Deadline  for  filing  in  the  first  simulation  (March  31,  1996) 

Invitation  to  open  calculation  of  problem  B,  (April  1996).  This  problem  provides  the  model  user  with  the  most  complete 
information  about  the  scenario,  including  the  results  of  experimental  tests  of  the  scenario.  Deficiencies  in  available  input 
(used  for  the  blind  calculation)  should  become  most  apparent  with  comparison  of  the  open  and  blind  calculation. 

Deadline  to  file  in  problem  B (June  31,  1996) 

Final  report  on  A,  and  B rounds  {see  above  for  a description}  (September  1996). 

Assessment  Rules 

The  simulation  code  assessment  includes  five  major  components,  which  occur  either 
explicitly  or  implicitly  during  the  process[6];  documentation,  verification,  performance 
validation,  sensitivity  study,  and  usability. 

Prerequisites  for  participation  into  the  round  robin  are: 

Full  written  description  of  the  code 

For  documentation  full  description  of  the  physical  basis  of  the  code,  the  main  structure  of  the  program,  required 
input  data,  produced  output  data,  and  user  instructions  should  be  written.  The  physical  basis  should  include  all 
mathematical  equations  used  in  the  code  either  fully  written  or  referenced  in  the  open  literature  in  a unique  way. 
Other  instructions  should  be  sufficiently  detailed,  that  an  educated  practitioner  in  the  field  should  be  able  to  use 
the  code  properly  following  instructions,  and  without  other  knowledge  of  the  code  (usability). 

Availability  of  the  source  code 

The  source  code  should  be  available  for  inspection  at  least  in  one  place  agreeable  to  peer  reviewers  outside  the 
institution  which  has  developed  the  code  (verification). 

Multiple  participation  in  the  round  robin. 
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For  a code  to  participate  in  the  round  robin  at  least  two  different  users  of  the  code  are  needed,  one  outside  the 
organization  where  code  is  developed.  These  users  should  simulate  the  given  problems  independently  (usability). 

The  First  Two  Standard  Problems 

SINGLE  PLUME  UNDER  A HOOD:  For  a rough  testing  of  numerics  smoke  filling  of  a 
hood  is  calculated  using  optional  plume  models,  and  compared  with  analytical  solutions. 
For  testing  the  sensitivity  on  the  initial  values,  several  input  data  sets  differing  from  each 
other  only  marginally  are  supplied.  For  scaling,  a small,  medium,  and  large  hood  are  to  be 
calculated. 

SINGLE  ROOM  WITH  A DOOR:  The  fire  scenario  consists  of  a single  room  with  a 
door,  and  a fire  load  of  wood  cribs.  Three  different  sizes  of  rooms  of  approximately 
similar  geometry  are  used. 

Comparison  of  Simulations 

For  verification,  a peer  review  will  be  carried  out,  listing  models  used.  It  is  felt  at  the 
moment,  a conclusive  review  prior  to  comparisons  of  calculations  would  be  practically 
impossible. 

During  this  first  phase  the  comparison  between  the  simulated  and  experimental  results  is 
made  by  inspection.  Qualitative  judgement  is  given  for  the  goodness  of  fit.  More  formal 
statistical  methods  may  be  tested,  and  used  if  practical. 

A workshop  on  simulations  has  been  arranged  for  September  1 996,  in  conjunction  with 
the  CIB/SFPE  workshop  in  Ottawa,  Canada  to  discuss  the  results  of  the  round  robin  before  the 
comparison  report  is  published  in  a final  form.  That  workshop  will  plan  the  continuation  of  the 
round  robin  exercise  for  the  next  scenarios  and  prepare  the  design  reports  for  the  precalculations 
of  those  scenarios. 


Related  Activities 

The  assessment  of  the  fire  simulation  codes  will  be  carried  out  in  cooperation  with  CIB 
W14  task  group  of  Engineering  Evaluation  of  Building  Fire  Safety  as  well  as  with 
ISO/TC92/SC4AVG1  and  other  organizations  related  to  the  work.  Since  the  data  to  carry  out 
detailed  evaluations  is  either  insufficient  or  nonexistent,  strong  interaction  is  needed  with 
organizations  and  laboratories  carrying  out  experiments.  The  following  activities  should  be 
discussed  to  find  proper  ways  of  carrying  them  out  outside  of  the  present  program  and  project. 

Use  accumulating  experience  of  round  robins  to  develop  rules  for  assessment  and  identify  areas  which  need 
additional  research. 
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Designing  and  carrying  out  all  scale  experiments  of  different  scenarios  to  assess  numerical  fire  codes.  There 
exists  only  a limited  number  of  relevant  experimental  data.  Design  of  an  experiment  should  be  based  on  a 
precalculated  scenario  to  define  measured  quantities  and  measurement  positions  in  a proper  way. 

Use  descriptions  of  the  codes  to  write  a handbook  on  simulation  codes. 


Conclusions 

The  development  of  performance-based  building  regulations  will  reduce  barriers  to 
international  trade  in  building  materials,  products,  design,  and  construction  by  providing 
objective  criteria  and  means  to  establish  compliance  with  these  criteria  that  are  not  proprietary. 
This  CIB  effort  is  central  with  regard  to  fire  regulations,  and  provides  key  support  and  pre- 
standardization research  to  the  International  Standards  Organization  effort  under  TC92/SC4  on 
Fire  Safety  Engineering. 
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